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(54) PROGRAM EVALUATION SYSTEM 

(57)Abstract: 

PURPOSE: To automatically decide a process result by generating 
simulation data which are not fixed (flexible evaluation of program to 
be evaluated). 

CONSTITUTION: A scenario storage part 20 stores a scenario 
determining the outline of the evaluation order of the program 100 to 
be evaluated. An input data storage part 202 stores input data sets 
corresponding to respective events. An expected value storage part 
205 stores expected value sets corresponding to the respective 
events. An execution means 201 generates simulation data by 
referring to the scenario storage part 204 and input data storage part 
202 and supplies the simulation data as an input to the program 100 to 
be evaluated. A decision means 203 compares a process result 
outputted from the program 100 to be evaluated with the expected 
value corresponding to the process result to decide whether or not 
the corresponding process result matches the expected value. 
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CLAIMS 



[Claim(s)] 

[Claim 1] The program for evaluation which inputs the data given from an external program, performs 
processing, and outputs a processing result to an external program again, The scenario storing section which 
stores the scenario which defines the outline of evaluation order, and the input data storing section which 
stores the input data set corresponding to each event, An activation means to generate simulation data with 
reference to the expected-value storing section which stores the expected-value set corresponding to each 
event, and said scenario storing section and said input data storing section, and to give the simulation data as 
an input of said program for evaluation, The program-evaluation method characterized by having a judgment 
means to be the processing result outputted from said program for evaluation, and the expected value under 
expected-value set of said expected-value storing circles, and to judge whether the expected value 
corresponding to the processing result concerned is compared, and the processing result concerned suits the 
expected value concerned. 

[Claim 2] The program-evaluation method according to claim 1 characterized by having said scenario storing 
section which stores the scenario which has serially the correspondence information on event generating time 
of day and an event number about each event. 



[Translation done.] 
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DETAILED DESCRIPTION 



[Detailed Description of the Invention] 
[0001] 

[Industrial Application] This invention relates to the program-evaluation method by which the program which 
operates on a computing system is evaluated (the program set as the object of evaluation is called "program 
for evaluation"). 
[0002] 

[Description of the Prior Art] Drawing 4 is the block diagram showing the configuration of an example of the 
conventional program-evaluation method. 

[0003] The conventional program-evaluation method shown in drawing 4 is constituted including the program 
400 for evaluation, and the simulation program 500 which has the activation means 501, the input data storing 
section 502, and the display means 503. 

[0004] The simulation program 500 simulates the external program which has an interface about an entry of 
data required in order that the program 400 for evaluation may perform processing, and an interface about the 
output of a processing result. 

[0005] Thus, the constituted conventional program-evaluation method was performing the following 
processings. 

[0006] The activation means 501 within the simulation program 500 pulls out each input data one by one from 
the input data group (fixed data constellation which has serially data used as the input of the program 400 for 
evaluation) stored in the input data storing section 502, and gives it to the program 400 for evaluation by 
making each input data into simulation data (the program 400 for evaluation inputs this simulation data). 
[0007] The program 400 for evaluation performs processing based on the simulation data, and outputs a 
processing result to the simulation program 500. 

[0008] The display means 503 within the simulation program 500 displays the processing result outputted from 
the program 400 for evaluation one by one (a hard copy output is included). 

[0009] Since the above processings were performed by the conventional program-evaluation method, whether 
the processing result of the program 400 for evaluation was right or it having mistaken, and the evaluation 
about the program for evaluation were left to the analysis by the analysis member based on the display by the 
display means 503. 
[0010] 

[Problem(s) to be Solved by the Invention] By the conventional program-evaluation method mentioned above, 
since the data (simulation data) which the program for evaluation inputs were fixed, they had the trouble that 
flexible evaluation of the program for evaluation became difficult. 

[001 1] Moreover, since it is [ that a processing result is only displayed and ] and people (analysis member) 
needed to analyze the processing result behind about evaluation of the processing result which the program 
for evaluation outputs, in order to evaluate the program for evaluation, many helps (effort) were required, and 
there was a trouble that it could not respond immediately about the error of the program for evaluation. 
[0012] It is for offering the program-evaluation method whose automatic judgment (evaluation) of a processing 
result the purpose of this invention becomes possible [ setting up the simulation data which is not fixed by 
installation of the scenario storing section (generation) ] in view of an above-mentioned point (flexible 
evaluation of the program for evaluation attained), and is attained by installation with the expected-value 
storing section and a judgment means. 

[0013] In addition, as a conventional technique about automation of evaluation of a program etc., "JP.2- 
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281339,A (automatic evaluation method of a data processor)" and "JP,62-239241,A (conversational-mode 
data-processing program automatic evaluation method)" are released. However, in these conventional 
techniques, a premise does not carry out "evaluation of the program for evaluation using a simulation 
program" like this invention, but this invention differs in a focus. 
[0014] 

[Means for Solving the Problem] The program for evaluation which the program-evaluation method of this 
invention inputs the data given from an external program, performs processing, and outputs a processing result 
to an external program again, The scenario storing section which stores the scenario which defines the outline 
of evaluation order, and the input data storing section which stores the input data set corresponding to each 
event, An activation means to generate simulation data with reference to the expected-value storing section 
which stores the expected-value set corresponding to each event, and said scenario storing section and said 
input data storing section, and to give the simulation data as an input of said program for evaluation, It has a 
judgment means to be the processing result outputted from said program for evaluation, and the expected 
value under expected-value set of said expected-value storing circles, and to judge whether the expected 
value corresponding to the processing result concerned is compared, and the processing result concerned 
suits the expected value concerned. 
[0015] 

[Example] Next, this invention is explained to a detail with reference to a drawing. 

[0016] Drawing 1 is the block diagram showing the configuration of one example of the program-evaluation 
method of this invention. 

[0017] The program-evaluation method of this example is constituted including the program 100 for evaluation 
which inputs the data given from an external program (it may be the program which operates on a computing 
system which may be the program which operates on the same computing system as the program 100 for 
evaluation, and is different), performs processing, and outputs a processing result to an external program 
again, and the simulation program 200 it is the program which evaluates in the program 100 for evaluation, and 
it is the program simulate an above-mentioned external program. 

[0018] An activation means 201 for the simulation program 200 to generate simulation data and to give the 
simulation data as an input of the program 100 for evaluation, The input data storing section 202 which stores 
the input data set corresponding to each event, A judgment means 203 to judge whether the processing result 
outputted from the program 100 for evaluation is compared with the expected value in the expected-value 
storing section 205 (the processing result concerned and corresponding expected value), and the processing 
result concerned suits the expected value concerned, It is constituted including the scenario storing section 
204 which stores the scenario which defines the outline of evaluation order, and the expected-value storing 
section 205 which stores the expected-value set corresponding to each event. 

[0019] Drawing 2 is a block diagram for explaining actuation of the program-evaluation method of this example 
at the time of the program 100 for evaluation inputting simulation data. 

[0020] Drawing 3 is a block diagram for explaining actuation of the program-evaluation method of this example 
at the time (when the simulation program 200 judges the processing result) of the program 100 for evaluation 
outputting a processing result. 

[0021] Next, actuation of the program-evaluation method of this example constituted in this way is explained. 
[0022] The actuation at the time of 1st the program 100 for evaluation inputting simulation data with reference 
to drawing 2 is explained. 

[0023] The scenario storing section 204 within the simulation program 200 stores beforehand the information 
which defines the outline of scenario evaluation order required for evaluation of the program 100 for 
evaluation. A scenario has serially the record (the correspondence information on event generating time of day 
and an event number) which consists of event generating time of day which shows the timing which a certain 
event (the event from which the program 100 for evaluation serves as an opportunity which inputs a series of 
data constellations, i.e., the event which draws the outline of the data input process by the program 100 for 
evaluation) generates, and an event number which are the information which identifies the event about some 
events. In addition, if the contents of the scenario are information which defines the outline of evaluation 
order, they will not be restricted to such a configuration. 

[0024] To the last, since such a scenario is the outline about evaluation order, i.e., the information which 
shows a framework, it becomes possible [ receiving one program 100 for evaluation and setting up the 
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scenario of the mode of shoes freely ]. Therefore, this example ****** can set up simulation data that it is 
not fixed and various by the program-evaluation method of this invention (generation). 

[0025] As for the input data storing section 202, only the number of events stores the set (input data set) of 
the input data corresponding to the event number described by the scenario. The input data is prepared 
serially and, as for the contents of the input data set, the time interval between input data (time interval to 
show the time of day given to the program 100 for evaluation by making each input data into simulation data) 
is also recorded. 

[0026] In addition, the contents of the input data set can be adjusted according to the contents of evaluation 

etc. It will contribute, in order that this may also set up the simulation data which is not fixed. 

[0027] In case the activation means 201 performs evaluation of the program 100 for evaluation, it reads the 

scenario (when there are two or more scenarios which receive the program 100 for evaluation, it is either of 

them) over the program 100 for evaluation from the scenario storing section 204. 

[0028] The activation means 201 carries out sequential recognition of each event serially based on the 

contents of the scenario concerned. That is, each record within the scenario concerned (correspondence 

information on event generating time of day and an event number) is read one by one. 

[0029] Furthermore, the activation means 201 reads the input data set corresponding to the event number 

from the input data storing section 202 by using the event number in the read record as a key. 

[0030] Then, the activation means 201 is given to the program 100 for evaluation, and is given to the program 

100 for evaluation one by one based on the time interval between the input data specified in the input data set 

by making each subsequent input data into simulation data according to event generating time of day by 

making the input data of the beginning in an input data set into simulation data (such processing is continued 

until the program 100 for evaluation finishes inputting the input data of the last in the input data set 

concerned). 

[0031] The activation means 201 repeats the above processings from the first event in the scenario 
concerned to the last event. 

[0032] The actuation at the time of 2nd the program 100 for evaluation outputting a processing result with 
reference to drawing 3 is explained. 

[0033] The expected-value storing section 205 within the simulation program 200 stores each expected-value 
set (set of each expected value corresponding to each processing result of the program 1 00 for evaluation 
based on each input data in the input data set) corresponding to each input data set in the input data storing 
section 202 beforehand. That is, each expected-value set corresponding to each event number corresponding 
to each input data set is stored. 

[0034] In case the judgment means 203 performs evaluation of the program 100 for evaluation over the "first 
input data set" concerning processing of the above-mentioned activation means 201, it receives the directions 
from the activation means 201, and reads the expected-value set corresponding to the first input data set 
from the expected-value storing section 205. 

[0035] If the program 100 for evaluation performs processing about the input data of the beginning in the first 
input data set and outputs the processing result to the judgment means 203 within the simulation program 
200, the judgment means 203 compares the processing result with predetermined expected value (expected 
value corresponding to the input data of the beginning of the expected value under expected-value set read 
previously). 

[0036] The judgment means 203 judges whether the processing result concerned suits the expected value 
concerned based on this comparison, and records the result of that judgment (this record is outputted by the 
simulation program 200 as contents of evaluation of the program 100 for evaluation). In addition, the mode of 
being judged with "the processing result concerned suiting the expected value concerned" if it is judged 
whether it belonging to the range in which expected value shall be "the information which shows the range of 
a fixed value", and the value of a processing result is shown with expected value as an example of the mode of 
this judgment and it belongs can be considered. 

[0037] Then, the judgment means 203 continues the same processing until it performs the judgment about the 
processing result of the processing about the input data after in the first input data set one by one and 
performs the judgment about the processing result corresponding to the input data of the last in the first input 
data set. 

[0038] Furthermore, after the judgment about the processing result group corresponding to the first input data 
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set is completed, the judgment means 203 reads the expected-value set corresponding to the following input 
data set from the expected-value storing section 205, and repeats the same processing as ****. And such 
processing is performed until the judgment about the processing result group corresponding to the last input 
data set is completed. 
[0039] 

[Effect of the Invention] As explained above, when evaluating the program for evaluation using a simulation 
program, this invention The data (simulation data) which the program for evaluation inputs are generated 
flexibly and variously according to a scenario. By performing evaluation of the program for evaluation by 
comparing the processing result based on the simulation data with the expected value under expected-value 
set It can carry out to a high speed and has automatic and the effectiveness that evaluation of a quality 
program is realizable in a short time for evaluation of the program for evaluation. 



[Translation done.] 
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TECHNICAL FIELD 



[Industrial Application] This invention relates to the program-evaluation method by which the program which 
operates on a computing system is evaluated (the program set as the object of evaluation is called "program 
for evaluation"). 
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PRIOR ART 



[Description of the Prior Art] Drawing 4 is the block diagram showing the configuration of an example of the 
conventional program-evaluation method. 

[0003] The conventional program-evaluation method shown in drawing 4 is constituted including the program 
400 for evaluation, and the simulation program 500 which has the activation means 501, the input data storing 
section 502, and the display means 503. 

[0004] The simulation program 500 simulates the external program which has an interface about an entry of 
data required in order that the program 400 for evaluation may perform processing, and an interface about the 
output of a processing result. 

[0005] Thus, the constituted conventional program-evaluation method was performing the following 
processings. 

[0006] The activation means 501 within the simulation program 500 pulls out each input data one by one from 
the input data group (fixed data constellation which has serially data used as the input of the program 400 for 
evaluation) stored in the input data storing section 502, and gives it to the program 400 for evaluation by 
making each input data into simulation data (the program 400 for evaluation inputs this simulation data). 
[0007] The program 400 for evaluation performs processing based on the simulation data, and outputs a 
processing result to the simulation program 500. 

[0008] The display means 503 within the simulation program 500 displays the processing result outputted from 
the program 400 for evaluation one by one (a hard copy output is included). 

[0009] Since the above processings were performed by the conventional program-evaluation method, whether 
the processing result of the program 400 for evaluation was right or it having mistaken, and the evaluation 
about the program for evaluation were left to the analysis by the analysis member based on the display by the 
display means 503. 
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EFFECT OF THE INVENTION 



[Effect of the Invention] As explained above, when evaluating the program for evaluation using a simulation 
program, this invention The data (simulation data) which the program for evaluation inputs are generated 
flexibly and variously according to a scenario. By performing evaluation of the program for evaluation by 
comparing the processing result based on the simulation data with the expected value under expected-value 
set It can carry out to a high speed and has automatic and the effectiveness that evaluation of a quality 
program is realizable in a short time for evaluation of the program for evaluation. 
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TECHNICAL PROBLEM 



[Problem(s) to be Solved by the Invention] By the conventional program-evaluation method mentioned above, 
since the data (simulation data) which the program for evaluation inputs were fixed, they had the trouble that 
flexible evaluation of the program for evaluation became difficult. 

[001 1] Moreover, since it is [ that a processing result is only displayed and ] and people (analysis member) 
needed to analyze the processing result behind about evaluation of the processing result which the program 
for evaluation outputs, in order to evaluate the program for evaluation, many helps (effort) were required, and 
there was a trouble that it could not respond immediately about the error of the program for evaluation. 
[0012] It is for offering the program-evaluation method whose automatic judgment (evaluation) of a processing 
result the purpose of this invention becomes possible [ setting up the simulation data which is not fixed by 
installation of the scenario storing section (generation) ] in view of an above-mentioned point (flexible 
evaluation of the program for evaluation attained), and is attained by installation with the expected-value 
storing section and a judgment means. 

[0013] In addition, as a conventional technique about automation of evaluation of a program etc., "JP,2- 
281339.A (automatic evaluation method of a data processor)" and "JP,62-239241,A (conversational-mode 
data-processing program automatic evaluation method)" are released. However, in these conventional 
techniques, a premise does not carry out "evaluation of the program for evaluation using a simulation 
program" like this invention, but this invention differs in a focus. 
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MEANS 



[Means for Solving the Problem] The program for evaluation which the program-evaluation method of this 
invention inputs the data given from an external program, performs processing, and outputs a processing result 
to an external program again, The scenario storing section which stores the scenario which defines the outline 
of evaluation order, and the input data storing section which stores the input data set corresponding to each 
event, An activation means to generate simulation data with reference to the expected-value storing section 
which stores the expected-value set corresponding to each event, and said scenario storing section and said 
input data storing section, and to give the simulation data as an input of said program for evaluation, It has a 
judgment means to be the processing result outputted from said program for evaluation, and the expected 
value under expected-value set of said expected-value storing circles, and to judge whether the expected 
value corresponding to the processing result concerned is compared, and the processing result concerned 
suits the expected value concerned. 
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EXAMPLE 



[Example] Next, this invention is explained to a detail with reference to a drawing. 

[0016] Drawing 1 is the block diagram showing the configuration of one example of the program-evaluation 
method of this invention. 

[0017] The program-evaluation method of this example is constituted including the program 100 for evaluation 
which inputs the data given from an external program (it may be the program which operates on a computing 
system which may be the program which operates on the same computing system as the program 100 for 
evaluation, and is different), performs processing, and outputs a processing result to an external program 
again, and the simulation program 200 it is the program which evaluates in the program 100 for evaluation, and 
it is the program simulate an above-mentioned external program. 

[0018] An activation means 201 for the simulation program 200 to generate simulation data and to give the 
simulation data as an input of the program 100 for evaluation, The input data storing section 202 which stores 
the input data set corresponding to each event, A judgment means 203 to judge whether the processing result 
outputted from the program 100 for evaluation is compared with the expected value in the expected-value 
storing section 205 (the processing result concerned and corresponding expected value), and the processing 
result concerned suits the expected value concerned, It is constituted including the scenario storing section 
204 which stores the scenario which defines the outline of evaluation order, and the expected-value storing 
section 205 which stores the expected-value set corresponding to each event. 

[0019] Drawing 2 is a block diagram for explaining actuation of the program-evaluation method of this example 
at the time of the program 100 for evaluation inputting simulation data. 

[0020] Drawing 3 is a block diagram for explaining actuation of the program-evaluation method of this example 
at the time (when the simulation program 200 judges the processing result) of the program 100 for evaluation 
outputting a processing result. 

[0021] Next, actuation of the program-evaluation method of this example constituted in this way is explained. 
[0022] The actuation at the time of 1st the program 100 for evaluation inputting simulation data with reference 
to drawing 2 is explained. 

[0023] The scenario storing section 204 within the simulation program 200 stores beforehand the information 
which defines the outline of scenario evaluation order required for evaluation of the program 100 for 
evaluation. A scenario has serially the record (the correspondence information on event generating time of day 
and an event number) which consists of event generating time of day which shows the timing which a certain 
event (the event from which the program 100 for evaluation serves as an opportunity which inputs a series of 
data constellations, i.e., the event which draws the outline of the data input process by the program 100 for 
evaluation) generates, and an event number which are the information which identifies the event about some 
events. In addition, if the contents of the scenario are information which defines the outline of evaluation 
order, they will not be restricted to such a configuration. 

[0024] To the last, since such a scenario is the outline about evaluation order, i.e., the information which 
shows a framework, it becomes possible [ receiving one program 100 for evaluation and setting up the 
scenario of the mode of shoes freely ]. Therefore, this example ****** can set up simulation data that it is 
not fixed and various by the program-evaluation method of this invention (generation). 

[0025] As for the input data storing section 202, only the number of events stores the set (input data set) of 
the input data corresponding to the event number described by the scenario. The input data is prepared 
serially and, as for the contents of the input data set, the time interval between input data (time interval to 
show the time of day given to the program 100 for evaluation by making each input data into simulation data) 



http:/ / www4.ipdl.ncipi.go.jp/cgi-bin/ tran_web_cgi_ejje 



6/20/2006 



JP,07-013809,A [EXAMPLE] 



Page 2 of 3 



is also recorded. 

[0026] In addition, the contents of the input data set can be adjusted according to the contents of evaluation 

etc. It will contribute, in order that this may also set up the simulation data which is not fixed. 

[0027] In case the activation means 201 performs evaluation of the program 100 for evaluation, it reads the 

scenario (when there are two or more scenarios which receive the program 100 for evaluation, it is either of 

them) over the program 100 for evaluation from the scenario storing section 204. 

[0028] The activation means 201 carries out sequential recognition of each event serially based on the 

contents of the scenario concerned. That is, each record within the scenario concerned (correspondence 

information on event generating time of day and an event number) is read one by one. 

[0029] Furthermore, the activation means 201 reads the input data set corresponding to the event number 

from the input data storing section 202 by using the event number in the read record as a key. 

[0030] Then, the activation means 201 is given to the program 100 for evaluation, and is given to the program 

100 for evaluation one by one based on the time interval between the input data specified in the input data set 

by making each subsequent input data into simulation data according to event generating time of day by 

making the input data of the beginning in an input data set into simulation data (such processing is continued 

until the program 100 for evaluation finishes inputting the input data of the last in the input data set 

concerned). 

[0031] The activation means 201 repeats the above processings from the first event in the scenario 
concerned to the last event. 

[0032] The actuation at the time of 2nd the program 100 for evaluation outputting a processing result with 
reference to drawing 3 is explained. 

[0033] The expected-value storing section 205 within the simulation program 200 stores each expected-value 
set (set of each expected value corresponding to each processing result of the program 100 for evaluation 
based on each input data in the input data set) corresponding to each input data set in the input data storing 
section 202 beforehand. That is, each expected-value set corresponding to each event number corresponding 
to each input data set is stored. 

[0034] In case the judgment means 203 performs evaluation of the program 100 for evaluation over the "first 
input data set" concerning processing of the above-mentioned activation means 201, it receives the directions 
from the activation means 201, and reads the expected-value set corresponding to the first input data set 
from the expected-value storing section 205. 

[0035] If the program 100 for evaluation performs processing about the input data of the beginning in the first 
input data set and outputs the processing result to the judgment means 203 within the simulation program 
200, the judgment means 203 compares the processing result with predetermined expected value (expected 
value corresponding to the input data of the beginning of the expected value under expected-value set read 
previously). 

[0036] The judgment means 203 judges whether the processing result concerned suits the expected value 
concerned based on this comparison, and records the result of that judgment (this record is outputted by the 
simulation program 200 as contents of evaluation of the program 1 00 for evaluation). In addition, the mode of 
being judged with "the processing result concerned suiting the expected value concerned" if it is judged 
whether it belonging to the range in which expected value shall be "the information which shows the range of 
a fixed value", and the value of a processing result is shown with expected value as an example of the mode of 
this judgment and it belongs can be considered. 

[0037] Then, the judgment means 203 continues the same processing until it performs the judgment about the 
processing result of the processing about the input data after in the first input data set one by one and 
performs the judgment about the processing result corresponding to the input data of the last in the first input 
data set. 

[0038] Furthermore, after the judgment about the processing result group corresponding to the first input data 
set is completed, the judgment means 203 reads the expected-value set corresponding to the following input 
data set from the expected-value storing section 205, and repeats the same processing as ****. And such 
processing is performed until the judgment about the processing result group corresponding to the last input 
data set is completed. 
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DESCRIPTION OF DRAWINGS 



[Brief Description of the Drawings] 

[Drawing 1] It is the block diagram showing the configuration of one example of the program-evaluation 
method of this invention. 

[Drawing 2] It is a block diagram for explaining actuation of the program-evaluation method shown in drawing 1 
at the time of the program for evaluation inputting simulation data. 

[Drawing 3] It is a block diagram for explaining actuation of the program-evaluation method shown in drawing 1 
at the time of the program for evaluation outputting a processing result. 

[Drawing 4] It is the block diagram showing the configuration of an example of the conventional program- 
evaluation method. 
[Description of Notations] 
100 Program for Evaluation 

200 Simulation Program 

201 Activation Means 

202 Input Data Storing Section 

203 Judgment Means 

204 Scenario Storing Section 

205 Expected-Value Storing Section 
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DRAWINGS 



[Drawing 1] 
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[Drawing 3] 



,100 



ft* 



[Drawing 4] 

<40 0 



■fay? a 



ft?) 



200 



201 



(205 



-203 



501 



503 



500 



1 



502 



Ailv — ^ 



[Translation done.] 



http://vmw4Jpdl.ncipi.gojp/cgi-bin/tran_web_cgLejue 



6/20/2006 



This Page is Inserted by IFW Indexing and Scanning 
Operations and is not part of the Official Record. 

BEST AVAILABLE IMAGES 

Defective images within this document are accurate representations of the original 
documents submitted by the applicant. 

Defects in the images include but are not limited to the items checked: 

□ BLACK BORDERS 

□ IMAGE CUT OFF AT TOP, BOTTOM OR SIDES 
□'faded TEXT OR DRAWING 

□Iblurred or illegible text or drawing 

□ skewed/slanted images 

□ color or black and white photographs 

□ gj6ay scale documents 
lines or marks on original document 

□freference(s) or exhibit(s) submitted are poor quality 

□ OTHER: 



IMAGES ARE BEST AVAILABLE COPY. 
As rescanning these documents will not correct the image 
problems checked, please do not report these problems to 
the IFW Image Problem Mailbox. 



